367 .39427X00 

IN THE UNITED STATES PATENT AND TRADEMARK OFFICE 

Applicants: P. NENONEN, et al 
Serial No.: 09/742,055 
Filed: December 22, 2000 

For: DIGITAL IMAGING 

Group: Not yet assigned 

Examiner: Not yet assigned 

LETTER CLAIMING RIGHT OF PRIORITY 

Assistant Commissioner April 17, 2001 

for Patents 
Washington, D.C. 20231 

Sir : 

Under the provisions of 3 5 USC 119 and 3 7 CFR 1.55, 

applicants hereby claim the right of priority based on: 

Great Britain Application No. 9 93 0409.9 
Filed: Deceinber 22, 1999 

A certified copy of said application document is attached 
hereto. 

Respectfully submitted. 



CIB/jdc 

Enclosures 

703/312-6600 



Carl I . Brun/didge 
Registration No. 29,621 

ANTONELLI, TERRY, STOUT & KRAUS , LLP 



The. % 




% Office I 




INVESTOR IN PEOPLE 

The Patent Office 
Concept House 
Cardiff Road 
Newport 
South Wales 
NPIO 8QQ 



I, the undersigned, being an officer duly authorised m accordance with Section 74(1) and (4) 
oftheDeregulation&Contracting Out Act 1994, to sign and issue certificates on behalf of the 
Comptroller-General, hereby certify that annexed hereto is a true copy of the documents as 
originally filed in connection with the patent application identified therein. 



In accordance with the Patents (Conq)anies Re-registration) Rules 1982, if a company named 
in this certificate and any accompanying documents has re-registered under die Companies Act 
1980 with the same name as that with which it was registered immediately before re- 
registration save for the substitution as, or inclusion as, the last part of the name of the words 
"public limited conq)any " or their equivalents in Welsh, references to the name of the conq)any 
in fliis certificate and any accompanying documents shall be treated as references to the name 
with which it is so re-registered. 

y 

In accordance with the rules, the words "public limited conq)any" may be replaced by p.Lc, 
pic, PX.C. or PLC, 



Re-registration under die Companies Act does not constitute a new legal entity but merely 
subjects the company to certain additional company law rules. 

CERTIFIED COPY OF 



mm DOCUMENl 







Signed 

Dated 29 November 2000 



ERTIFIED COPY OF 
'PIIY DOCUMINT 



An Executive Agency of the Department of Trade and Industry 



Patents Form 1/77 

Pa 



ct 197J 



QflSoe 




Request 

(See the notes oj 
explanatory it 
this form) 




24DEC99 E501555-i D02716- 
POl/7700 0.00-9930409.9 



The Patent Office 

Cardiff Road 
Newport 
Gwent NP9 IRH 



1. Your reference , ■ 


PAT 99432 GB 


2. Patent application number 


9930409.9 


(The Patent Office wiUfUl in this part) 



3. Full name, address and postcode of the or of 
each applicant (underline all surnames) 



Patents ADP number (if you know it) 

If the applicant is a corporate body, give the 
country/state of its incorporation 



NOKIA MOBILE PHONES LIMITED 
KEILALAHDENTIE 4 
02150 ESPOO 
FINLAND 



FINLAND 



4. Title of the invention 



DIGITAL IMAGING 



5- Name of your agent (if you have one) 

"Address for service" in the United Kingdom 
to which all correspondence should be sent 
(including the postcode) 



Patents ADP nimiber (if you know it) 



NOKIA IPR DEPARTMENT 
NOKIA HOUSE 
SUMMIT AVENUE 
FARNBOROUGH 
HAMPSHIRE 
GU140NG UK 



6. If you are declaring priority from one or more 
earlier patent applications, give the country 
and the date of filing of the or of each of these 
earlier applications and (if you know it) the or 
each application number 



Country Priority application number 

(if you know it) 



Date of filing 
(day / montb /year) 



7. If this application is divided or otherwise 
derived from an earlier UK application, 
give the number and the filing date of 
the earlier application 



Number of earlier application 



Date of filing 
(day / month /year) 



8. Is a Statement of inventorship and of right 
to grant of a patent required in support of 
this request? (Answer ^Yes' if 

a) any applicant named in part 3 is not an inventor, or 

b) there is an inventor who is not named as an 
applicant, or 

c) any named applicant is a corporate body 
See note (d)) 



YES 



Patents Fonn 1/77 



Patents Form 1/77 



9. Enter the number of sheets for any of the 
foUowing items you are filing with this form. 
Do not count copies of the same document 

Continuation sheets of this form 
Description 

Abstract 



20 

2 
1 



Drawingr^; 



10. If you are also filing any of the following, 
state how many against each item. 



Priority documents 

Translations of priority documents 

Statement of inventorship and right 
to grant of a patent (Patents Form 7/77) 

Request for preliminary examination 
and search (Patents Form 9/77) 



15 -i- 



Request for substantive examination 

(Patents Form IO/77) 

Any other documents 

(please specify) 



11. 




I/We request the grant of a patent on the basis of this application. 
Sign^a|p^^e , Date 



12. Name and daytime telephone number of 
person to contact in the United Kingdom 



DHIREN PATEL 0125R 865285 



Warning 

After an application for a patent has been filed, the Comptroller of the Patent Office will consider whether publication 
or communication of the invention should be prohibited or restricted under Section 22 of the Patents Act 1977. You 
will be informed if it is necessary to prohibit or restrict your invention in this way Furthermore, if you live in the 
United Kingdom. Section 23 of the Patents Act 1977 stops you from applying for a patent abroad without first getting 
tvrinen permission from the Patent Office unless an application has been filed at least 6 weeks beforehand in the 
united Kingdom for a patent for the same invention and either no direction prohibiting publication or, 
communication has been given, or any such direction has been revoked. 

Notes 

a) If you need help to fill in this form or you have any questions, please contact the Patent Office on 0645 500505. 

b) Write your answers in capital letters using black ink or you may type them. 

c) If there is not enough space for all the relevant details on any part of this form, please continue on a separate 
^^^^^ of paper and write "see continuation sheet" in the relevant part (s). Any continuation sheet should be 
attached to this form. 

d) If you have answered 'Yes' Patents Form 7/77 will need to be filed. 

e) Once you have filled in the form you must remember to sign and date it 

f) For details of the fee and ways to pay please contact the Patent Office, 



Patents Form 1/77 



Digital imaging 

The present invention relates to digital imaging technology, and more 
5 particularly to contrast and edge enhancement in still and moving digital 
imaging. 

Digital imaging offers a great many advantages over traditional analogue 
systems, supporting services such as video telephony and multi-media 
10 applications. Accordingly, digital video imaging is currently the subject of 
extensive research. 

One particular area of digital imaging research is that of image enhancement. 
In digital imaging, there is need for image enhancement because images 

15 typically suffer from various types of degradation, such as lack of sharpness, 
noise, compression artefacts, and poor contrast and brightness. Mobile 
imaging products presently under development are especially prone to such 
degradations due to the low performance of digital camera devices and 
narrow data transmission bandwidth which requires the use of high 

20 compression ratios. In digital camera products, the poor image quality arises 
from shortcomings in fundamental sensor properties: the lack of contrast 
typically arises from the sensor itself, while the lack of sharpness is the result 
of both low quality optics and low resolution sensor typical of imaging systems 
that are likely to be characteristic of forthcoming first generation mobile 

25 telecommunication terminals with image presentation and manipulation 
functionality. Thus, there is a need for methods to enhance image contrast 
and sharpness, and in current digital image enhancement techniques this is 
commonly carried out by concealing artefacts and reconstructing degraded 
parts of an image. 

30 
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Image enhancement principally concerns accentuation of image features such 
as edges, boundaries, or contrast. This makes image details more visible and 
images look subjectively sharper or crisper without appearing unnatural 
therefore making a graphic display more useful for display and analysis. 
5 Whilst the enhancement process does not increase the inherent information 
content in the data, it does increase the dynamic range of chosen features so 
that they can be more easily detected. 

Image enhancement techniques can be divided roughly into three sub-classes 
10 comprising point, spatial, and transform operations. 

Point operations are performed on individual pixels and as such are 
independent of surrounding pixel values. Pixel mapping operations are one 
example of point operations. Pixel mapping involves mapping a pixel value to 
15 a new value with a function. For example: 

where 

lyj = pixel's coordinates 
y = output pixel 
X ~ input pixel 

and f is a function relating x and y. 

Spatial operations are performed on groups of pixels. Certain image 
20 smoothing and filtering functions are spatial operations where, for example, a 
pixel's value may be modified by taking an average of surrounding pixel 
values. 

Transform operations convert image data from pixel values into another form, 
25 for example spatial frequency components. Discrete cosine transforms (DCT) 
and wavelet transforms are examples of transform operations commonly used 
in image processing applications. 
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Examples of enhancement techniques in the sub-classes outlined above 
include techniques such as grey level and contrast manipulation, noise 
reduction, edge crispening and sharpening, filtering, interpolation and 
magnification, and pseudo-colouring. 

5 

In practice, images taken by a digital camera normally suffer from smoothness 
problems irrespective of the pixel resolution. This means that there is always 
likely to be a need for image sharpening techniques. Image sharpening can 
be performed by applying various forms of contrast enhancement 

10 (stretching/shrinking) and edge enhancement techniques. Contrast 
enhancement is a point operation that can be used to enhance images taken 
in poor or non-uniform lighting conditions, or acquired with an imaging sensor 
having a small dynamic range. Contrast manipulation also tends to alter the 
visibility of edges, which changes the perceived sharpness of the image. 

15 Edge enhancement techniques are typically spatial operations and they 
enhance images that suffer from overall smoothness. A technique known as 
unsharp masking is an example of an edge enhancement technique. 

Referring first to contrast enhancement, a well known method for enhancing 
20 contrast is histogram-based pixel mapping. In this method a pixel-mapping 
function is formed utilising a histogram of the image or a part of an image. 
The histogram contains the number of pixel occurrences at each level of the 
input signal range. The mapping function is derived from a cumulative 
histogram so that it has high slope corresponding to high values, in the 
25 histogram. The cumulative histogram is typically formed using an equation 
having the form: 

Jfc=0 

where 

C- = bin i of cumulative histogram 
~ bin k of histogram 
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Note, a bin is a pixel value range, that may cover a single pixel value or nriore 
than one pixel value. 

In the basic method the mapping function is: 

The mapping function or the cumulative histogram is typically scaled so that 
the mapping covers substantially exactly the available dynamic range, i.e.: 
fiL) = L, 
where 

L = largest value of the available dynamic range 

The scaling is achieved, for example, using equation: 

10 f{x)^L^ 

The above equations can be found in slightly varying forms in many signal 
processing textbooks. For example, in one alternative approach the first 
nonzero bin is subtracted from the histogram before or during computation of 
the cumulative histogram. If such a subtraction is not performed the darkest 
1 5 pixel value of the image is not mapped to zero (Cq^O). Other modifications 
also exist. 

The above equations are usable substantially only if there are as many bins 
as signal levels, for example, 256 bins are required for a signal with 8 
20 bits/sample. If fewer bins are used, each bin contains pixels belonging to a 
certain signal range. The above equations provide mapping function, values 
at the borders of bins. Additional values may be obtained by interpolation. 

Examples of a histogram, a cumulative histogram and corresponding mapping 
25 function are presented in Figure 1. 

Using histogram-based pixel mapping techniques, the contrast of an image 
can be increased by mapping input pixel values to output pixel values with a 



linear function having a slope (i.e. derivative) greater than one. A slope less 
than one decreases contrast while a slope equal to one produces no change 
in contrast. The mapping function can also be non-linear, having a varying 
slope. In this case contrast around some pixel values of the image is 
5 increased, the contrast in other pixel value ranges may be decreased or not 
altered at all. An example of a non-linear mapping function is shown in Figure 
2. 

An acknowledged limitation of histogram-based pixel mapping is that it tends 
10 to make noise and other artefacts disturbingly visible. Because increasing 
contrast increases the visibility of variations in the input signal, if there is noise 
present in the input signal, its visibility is also increased. Because of this, 
histogram-based pixel mapping is often considered unsatisfactory for contrast 
enhancement of natural images. 

15 

Nevertheless, histogram-based pixel mapping is the basis for a substantial 
number of contrast enhancement algorithms. So whilst these algorithms form 
a mapping function based on the image histogram, attempts are made to 
mitigate the unwanted effects of the basic mapping procedure using 
20 modifications and additions to the basic algorithm. These modifications 
include, for example, reshaping the histogram, reshaping the mapping 
function, filtering the image, the histogram or the mapping function and 
limiting the maximum value of the histogram. 

25 Turning now to edge enhancement, specifically in the context of unsharp 
masking, this is a spatial filtering technique for improving the visibility of edges 
and thus improving perceived image quality. Unsharp masking is commonly 
implemented in the form of a highpass filter in series with an adjustable gain 
block, as shown in Figure 3. In the illustrated arrangement, a highpass 

30 filtered and scaled image signal is added to the original image signal as 
shown in Figure 4a. Thus, the desired improvement is achieved by 
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emphasising high frequency components of the image. Specifically, the 
highpass filter represents a second order derivative of the original signal, as 
represented in Figure 4b. When the highpass filtered signal is added back to 
the original signal, it creates under and overshoots around edges as shown in 
5 Figure 4c. The human visual system itself creates similar under and 
overshoots around sharp edges. This phenomenon is known as the Mach 
band effect. It demonstrates that the human visual system perceives edges in 
a non-linear manner, and that perception depends not only on luminance but 
also on image structure. For this reason, the human visual system perceives 
10 edges with added under and over shoot such as those produced by unsharp 
masking, as sharper edges. 



However, unsharp masking also tends to produce some undesirable effects. 
The presence of a highpass filter makes the system sensitive to noise and 
15 compression artefacts. Thus, unsharp masking typically produces 
perceivable artefacts, which are especially visible in uniform areas of an 
image. 

Accordingly, it has been recognised that in order to improve image quality it 
20 would be appropriate to modify an input signal of the highpass filter, especially 
when there is a noise component present in an image. One such 
modification, known in the art, is to use a standard predefined or fixed pixel 
mapping function prior to unsharp masking. Using a conventional fixed pixel 
mapping function, it is possible to emphasise certain edges whilst .limiting 
25 unwanted noise enhancement. Figure 5 depicts an unsharp masking scheme 
that utilises pixel mapping prior to highpass filtering. In the illustrated 
arrangement, a pixel mapping function is applied to the image before unsharp 
masking in order to limit the unwanted enhancement of noise components 
that occurs as a result of high-pass filtering. 



30 
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Figure 6 shows two possible pixel mapping function curves suitable for use 
prior to unsharp masking. Curve 1 suppresses high and low level signals and 
stretches mid-range pixel values. Accordingly, in the highpass filter high and 
low level edges are suppressed and mid level edges are enhanced. In Curve 
5 2 mid level edges are suppressed and high and low level edges are 
enhanced. In this way, the effect of using pixel mapping prior to highpass 
filter is to reduce noise. 

The effect of a function such as Curve 1 shown in Figure 6 is illustrated in 

10 Figure 7. This shows the effect of applying a conventional fixed pixel mapping 
function to an image that contains a single 'edge'. Here the edge is a 
transition from dark to light i.e. from very low intensity pixel values to very high 
intensity pixel values. In other words, the image contains just two regions, 
one of low luminance values and one of high luminance values, as shown in 

15 Figure 7a. Figure 7b presents a graph of pixel values created by scanning 
across a row of pixels from left to right. It is assumed that the two regions, 
which should be of essentially uniform intensity, exhibit some degree of noise. 
The noise manifests itself as small variations in pixel value (as can be seen 
from the Yipple' in Figure 7b). By applying the fixed pixel mapping function 

20 illustrated in Figure 7c, the noise in the image of Figure 7b can be suppressed 
effectively. The result of the fixed pixel mapping function is illustrated in 
Figure 7d. The way in which the noise is suppressed can be understood by 
examining the pixel mapping function in detail. It should be appreciated that 
the pixel mapping function describes the way in which pixel values, in the 

25 original image (i.e. that described by Figure 7b) are converted into pixel 
values in the contrast enhanced (adjusted) image, whose pixel value profile is 
illustrated in Figure 7d. Considering first reduction of noise in the low intensity 
region of the image, the relevant part of the pixel mapping function is that 
labelled W in Figure 7c. This part of the pixel mapping function determines 

30 how low intensity pixel values in the original image are translated into low 
intensity pixel values in the contrast enhanced image. Here the pixel mapping 
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function has a gradient less than 1, indicating that variations in pixel value will 
be reduced (or 'compressed*) by applying the function. The same is true in 
region of the pixel mapping function of Figure 7c. This part of the curve 
determines the way in which high intensity pixel values in the original image 
5 are translated into high intensity pixel values in the contrast-enhanced image. 
Thus, noise is effectively reduced at both extremes of the range. Referring to 
how mid-range intensity values are converted by the pixel mapping function in 
this example, the part of the function labelled has a gradient greater than 1 
and therefore, the range of pixel value variation in this region is actually 

10 enhanced (expanded or stretched), changing the profile of the transition 
between low and high intensity values. However, this effect cannot be 
considered as a true sharpening of the transition, such as that provided by 
unsharp masking. Contrast expansion has the effect of enhancing the 
difference between nearly identical mid-range pixel values and inevitably 

1 5 enhances the effect of noise in the pixel value range around the transition. 

The use of conventional fixed pixel mapping functions in order to condition an 
image prior to unsharp masking has certain limitations. These will be 
described with reference to Figure 8, which illustrates the effect of using the 

20 same pixel mapping function, introduced above, to enhance the contrast of a 
second example image. As in the first example, the image considered in 
Figure 8 is a very simple one, comprising two substantially uniform regions, 
one darker than the other (Figure 8a). The difference in intensity values 
between the two regions is less than in the previous example, although the 

25 same kind of noise is present (Figure 8b). The effect of the fixed pixel 
mapping function is illustrated in Figure 8d. While noise in the darker region 
is suppressed, noise in the lighter region is actually enhanced. This effect 
occurs because the pixel values of the lighter region correspond to a region of 
the pixel mapping function that has a gradient greater than one. Thus, small 

30 differences in pixel value are enhanced, rather than suppressed, causing an 
increase in noise. 
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It is common that images are processed using both (either fixed or histogram- 
based) pixel mapping contrast algorithms and unsharp masking edge 
enhancement algorithms. Referring to Figure 9, the combination of histogram 
5 -based pixel mapping and unsharp masking can be implemented by placing 
the histogram-based pixel mapping in the original image signal branch before 
(Option A), after (Option 0), or in parallel (Option B) with the edge 
enhancement unit. Commonly, image processing applications, such as 
certain commercially available software packages, implement options A or C, 

10 giving rise to a cascaded structure. However, it has been found that the best 
results are achieved by option B, in which the histogram-based pixel mapping 
unit is placed inside the unsharp masking unit, that is in the original image 
branch parallel to the highpass filter and not in cascade with it as in options A 
and C. In option B, any unwanted noise component that is emphasised by 

15 contrast enhancement, is not highpass filtered and thus not further 
emphasised. In both options A and C the second operation in the cascaded 
structure further emphasises noise artefacts introduced by the first operation. 
Here, the second operation refers to the latter operation, i.e. in option A, the 
second operation is unsharp masking, and in option C the second operation is 

20 histogram-based pixel mapping. Thus, the cascaded approach has the 
disadvantage that noise is 'doubly' enhanced. This creates very annoying 
noise structures, especially when a JPEG or DOT based hybrid codec has 
been used to compress the original source image. For example, assuming 
option A is used, initial contrast enhancement causes a certain increase in 

25 pixel value noise. This is further exaggerated by the following step of edge 
enhancement using e.g. unsharp masking. In the case of option C, because 
the unsharp masking emphasises edges it also heightens noise around 
edges. The amplitude of this noise is then further increased by the histogram- 
based contrast enhancement. Also, other artefacts caused by the unsharp 

30 masking, for example ringing effects may become more visible. The 
approach illustrated by option B, on the other hand, avoids double noise 
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enhancement. Here, contrast and edge enhancennent are performed in 
parallel and the high pass filtered image is added to the contrast enhanced 
image (as indicated by the addition symbol in Figure 9). 

5 Nevertheless, the combined image enhancement options of Figure 9 suffer 
from the problems associated with contrast and edge enhancement previously 
described above. 

Against this background, and in one aspect, the present invention resides in a 
10 method for image enhancement of a digital image comprising a set of image 
pixels, the method comprising applying an inverse histogram based mapping 
function to the image pixels. 

The inverse histogram-based pixel mapping can be applied alone or to any of 
15 options A, B and C described above. In a particularly preferred embodiment, 
inverse histogram-based pixel mapping is applied to option B, that is to say 
within an unsharp masking unit. 

in a preferred form of this aspect of the invention, the inverse histogram- 
20 based pixel mapping step is combined with an edge enhancement step such 
as unsharp masking. In such an arrangement the inverse histogram-based 
pixel mapping step improves the performance of the unsharp masking step. It 
also has an advantageous effect on combinations of contrast enhancement 
and unsharp masking techniques. 

25 

As explained earlier, unsharp masking techniques augment the subjective 
sharpness of an image by adding overshoot and undershoot components to 
edges within an image. The effect of inverse histogram-based pixel mapping 
is to change the pixel values according to their frequency of occurrence. In 
30 embodiments in which the inverse histogram-based pixel mapping step is 
combined with an edge enhancement step such as unsharp masking, the role 




f'"". 

11 

of the inverse histogram based pixel mapping is to act as a pre-processor for 
the high pass filter unit inside the edge enhancement unit, that alters pixel 
values in such a way that noise enhancement introduced by subsequent edge 
enhancement is reduced. 

5 

This arrangement serves to minimise the amplificaton of noise components 
while desired signal components are enhanced. More specifically, inverse 
histogram-based pixel mapping produces a reduction in contrast at signal 
ranges containing a large number of pixels and an increase in contrast 
10 elsewhere. Thus, noise is suppressed in e.g. uniform areas, where it is 
subjectively most visible. It has been found that the effect is substantially the 
same when inverse histogram-based pixel mapping is applied to: a) a large 
uniform area, and b) an area containing a large number of pixels having a 
small number of different values (e.g. a speckled region). 

15 

The first aspect of the invention includes a device, such as a portable radio 
communication device, capable of performing said image enhancement 
method, and extends to a software program product stored on a medium and 
configured to perform said image enhancement method. 

20 

In a second aspect of the invention, there is provided a method for image 
enhancement of a digital image comprising a set of image pixels, the method 
comprising applying in a loop of an edge enhancement unit that is parallel to 
the original image branch a histogram-based pixel mapping function ,to said 
25 set of image pixels prior to applying a highpass filter in said loop. 

As described earlier, use of a fixed pixel mapping function in connection with 
unsharp masking prior to high-pass filtering is known as illustrated in Figure 5. 
This has the disadvantage that such a function can only be chosen to 
30 enhance contrast in certain pixel value ranges and its effect on noise 
components in the image varies according to the nature of the image. 
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Accordingly, the inventors have noted that because the function illustrated in 
Figure 6 is constructed artificially it cannot be altered to adapt to image 
content and is therefore not optimal for all images. 

5 In contrast, the inventors have recognised that because a histogram-based 
pixel mapping function is constmcted with reference to the image itself, it is 
adaptive to image content and hence some of the problems relating to noise 
enhancement can be reduced. 

10 Used in conjunction with edge enhancement, the histogram can be adapted 
such that once a histogram of pixel values within an image has been 
determined, the profile of the histogram may be altered in order to change the 
nature of the pixel mapping function created from it. 

15 Referring now to the accompanying drawings, Figures 1 to 9 are used in 
conjunction with an account of the prior art in which: 

Figure 1 shows a histogram and corresponding mapping function; 

20 Figure 2 illustrates a non-linear mapping function; 

Figure 3 shows a block diagram of an unsharp masking scheme; 

Figure 4 illustrates the effect of an unsharp masking scheme on an edge of an 
25 image signal; 

Figure 5 shows an unsharp masking scheme with a fixed pixel-mapping unit 
prior to a high pass filter of the unsharp masking scheme; 

30 Figure 6 illustrates two possible curves of a pixel mapping function of figure 5; 
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Figure 7 shows the fixed pixel mapping function applied to an edge in a first 
noisy image; 

Figure 8 shows a fixed pixel mapping function applied to an edge in a second 
5 noisy image; 

Figure 9 shows possible places for a contrast enhancement unit. 

The present invention will now be described, by way of example only, with 
10 reference to the accompanying drawings, in which: 

Figure 10 illustrates one embodiment of the first aspect of the present 
invention; 

15 Figure 11 illustrates an inverted histogram constructed according to the 
embodiment of Figure 1 0; 

Figures 12, 13 and 14a-e show alternative arrangements of the first aspect of 
the present invention; and 

20 

Figure 15 shows an embodiment according to a second aspect of the 
invention. 

The first aspect of the present invention resides in the provision of aa ipverse 
25 histogram-based pixel mapping technique for enhancing a digital image. 
Whilst this invention can be used in isolation to operate on an image, it has 
been found that it provides particularly beneficial results when used with any 
image processing function that tends to increase noise visibility. The 
approach of the present invention, as well as being a method, resides also in 
30 a device that includes means for applying inverse histogram-based pixel 
mapping to a digital image. In a specific context, the device may be a mobile 
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telecommunications terminal, such as a mobile phone, that has the 
conventional radio frequency, hardware and software means normally 
required of a mobile phone and that is equipped with image processing 
capability and/or image capturing and display means. Alternatively, the 
5 device may simply be a computer equipped with digital image processing 
means implementing the method according to the invention. In the former 
case, the mobile terminal is able to make use of the present invention prior to 
transmission of a captured image, and/or for processing received images for 
display on the mobile terminal. 

10 

Referring initially to Figure 10. there is shown one embodiment of an inverse 
histogram-based pixel mapping unit that is provided as a pre-conditioning unit 
for a highpass filter unit inside an unsharp masking unit. The inverse 
histogram-based pixel mapping unit is disposed in series with the high pass 
15 filter that is inside the edge enhancement unit and both are in a branch or 
loop that is parallel to the original image signal branch. 

The inverse histogram-based pixel mapping function can be formed, for 
example, by inverting an image histogram and then subtracting the inverted 

20 histogram from the maximum value of the original histogram. This method is 
illustrated in Figure 11. A standard histogram is first obtained from an image 
and the maximum value of the histogram is determined. Then, every value in 
the histogram is subtracted from this maximum value. Next, a cumulative 
histogram is created. Finally, the inverse histogram-based pixel mapping 

25 function is derived from the cumulative histogram. After inversion, the 
mapping function is made as in the conventional histogram-based pixel 
mapping method. It should be appreciated that the inverse histogram-based 
pixel mapping function is not an inverse function in a strict mathematical 
sense (i.e. AvW^'/^W)- 
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Thus in one way the formation of the inverse histogram-based mapping 
function comprises the steps of; 

constructing a histogram of image pixel values, optionally processing the 
histogram, making an inverse histogram from the histogram, optionally 
5 processing the inverse histogram, making a cumulative inverse histogram 
from the inverse histogram, optionally processing the cumulative inverse 
histogram, making an inverse histogram-based pixel mapping function from 
the inverse histogram, optionally processing the inverse mapping function, 
and mapping pixel values using the inverse histogram-based pixel mapping 
10 function. 

The "inversion" can be performed on the cumulative histogram, or 
alternatively it can be applied to the mapping function. For example, 
formation of an inverse histogram-based pixel mapping function may 

1 5 comprise the steps of: 

constructing a histogram of image pixel values, optionally processing the 
histogram, making a cumulative histogram, optionally processing the 
cumulative histogram, making a mapping function from the cumulative 
histogram, optionally processing the mapping function, making an inverse 

20 mapping function, optionally processing the inverse mapping function and, 
mapping pixel values using the inverse histogram-based pixel mapping 
function. 

In constructing a histogram it is possible that regional histograms could be 
25 used, i.e. a histogram representing a localised portion of the image. The 
original histogram may be processed or modified before constructing an 
inverse. Modifications can also be applied to the inverted histogram. In 
addition, it is also possible to make an inverse mapping function from a 
mapping function derived from the original histogram. 
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Because the inverse histogram-based pixel mapping function is derived from 
the inverse histogram via an inverse cumulative histogram, and the 
conventional (non-inverted) histogram-based pixel mapping function is 
derived from an image histogram via a cumulative histogram, the inverse 
5 histogram-based pixel mapping function and conventional histogram-based 
pixel mapping function share a relationship with one another. Therefore, as 
explained above, there are various ways to construct an inverse histogram- 
based pixel mapping function from a conventional histogram-based pixel 
mapping function or from a cumulative histogram. For example, if the 
10 conventional histogram-based mapping function is differentiated, the 
derivative is inverted (in this context the inversion means subtraction from the 
maximum value) and the result is integrated, the obtained mapping function is 
the inverse histogram-based pixel mapping function. 

15 In an alternative approach, the inverse histogram-based pixel mapping 
function is produced by subtracting each value of a cumulative histogram 
(formed in a conventional manner) from a value larger than the maximum 
histogram value. Furthermore, adding an offset to the maximum value allows 
limitation of the maximum dynamic range suppression (no zero slopes). Other 

20 modifications can also be applied to the inverted histogram prior to formation 
of the inverse histogram-based pixel mapping function. What should be noted 
is the principal shape of the inverted histogram compared to the original. 
Peaks in the original histogram are transformed to troughs in the inverse 
histogram and vice versa. In mapping function terms this means that.r^gio^s 

25 of high-gradient are transformed to regions of low-gradient and vice versa. 

Accordingly, the inverse histogram-based pixel mapping function is formed so 
that it has a high slope corresponding to low values in the image histogram 
(i.e. pixel value ranges with a small number of occurrences) and a low slope 
30 corresponding to high values in the image histogram (i.e. pixel value ranges 
with a large number of occurrences). The effect of mapping the input pixel 
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values to the output pixel values with this kind of function is that contrast is 
reduced in signal ranges containing a large number of pixels and increased 
elsewhere. This kind of contrast manipulation decreases the visibility of small 
variations and noise in the statistically most important signal ranges. 

5 

It has been found that by applying a pixel mapping function constructed from 
an inverse histogram of the image to the image prior to the edge 
enhancement unit of an unsharp masking block, the problem of amplification 
of noise ordinarily caused by unsharp masking is significantly reduced. Noise 
10 amplification is particularly disturbing when a JPEG or DCT based hybrid 
codec has compressed the source image, and the present invention has been 
found to be particularly useful when applied to such images. 

Because inverse histogram-based pixel mapping reduces contrast and noise 
15 visibility in statistically important signal ranges, noise enhancement produced 
by unsharp masking is also reduced at these signal ranges. This has an 
advantageous effect, for instance, when the image contains a relatively wide 
and smooth area. Such an extensive smooth area gives rise to a 
concentration of high values in the image histogram. Inverse histogram- 
20 based pixel mapping has the effect of reducing noise amplification due to 
subsequent unsharp masking, particularly in such a large and smooth area. 
Elsewhere the unsharp masking sharpens edges without restrictions. Thus, 
while edges are suppressed in one part of the dynamic range e.g. for pixel 
values representing a large smooth area they are enhanced elsewhere in the 
25 dynamic range. Therefore "real edges" between regions are enhanced 
without restrictions. 

As previously explained, image processing operations often include both 
unsharp masking and a histogram based pixel-mapping algorithm for contrast 
30 enhancement. In this case, it has been found that the present invention 
significantly improves the resulting picture quality. Inverse histogram-based 
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pixel mapping reduces noise amplification in tlie areas of the image that are 
emphasised by the contrast enhancement algorithm. Due to this feature, both 
algorithms do not enhance the noise. 

5 Figure 12 shows an example of an Implementation of the present invention in 
conjunction with conventional histogram-based pixel mapping contrast 
enhancement and unsharp masking. The conventional histogram-based pixel 
mapping contrast enhancement unit is placed in the original image branch of 
the unsharp masking unit, and the combination of the inverse histogram- 
10 based pixel-mapping unit and the unsharp masking unit is placed in a loop 
that is parallel to the original image branch. The output of the parallel loop is 
added to the histogram-based pixel mapping contrast enhancement unit 
output. 

15 Figure 13 shows a modified arrangement in which an additional (low-pass) 
filtering block added in the original image branch (into the upper path), after 
the contrast enhancement operation. The purpose of the filtering block is to 
reduce the noise introduced by conventional histogram-based contrast 
enhancement. 

20 

A second aspect of the invention is depicted in Figure 15 in which there is 
provided a histogram-based pixel mapping unit alongside and prior to an 
unsharp masking unit in a loop of an image enhancement arrangement, the 
loop being parallel to the original image branch. The output of the. parallel 
25 loop is added to the original image. With this arrangement because the 
mapping function adapts with the image content, noise enhancement has 
been found to be reduced. 

The present invention may also be applied for enhancement of digital video 
30 sequences, in which case it is advantageous to add temporal processing to 
the histogram. A video signal consists of a sequence of digital images. Each 
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image can be enhanced individually, but the same histograms cannot be used 
for two images that have different contents. This means that histograms are 
preferably gathered for each image. However, processing images in the 
video sequence individually produces disturbing flicker, because even quite 
5 unnoticeable visual differences can cause two images to have substantially 
different histograms. Therefore, temporal processing, for example low pass 
filtering, of the histograms is needed. Typically this kind of processing is 
adaptive to sudden large changes in image characteristics, for example due to 
scene-cuts. 

10 

The present invention may be embodied in other specific forms without 
departing from its essential attributes. The inverse histogram-based pixel 
mapping function of the present invention has been described above in the 
context of a pre-processor for an edge enhancement function. However, the 

15 present invention may be used on its own or in a variety of different ways. 
Figure 14 illustrates a number of different applications of an inverse 
histogram-based mapping function unit. Figure 14a shows the inverse 
histogram-based mapping unit applied on its own to an image. Figure 14b 
shows the inverse histogram-based mapping unit as a post-processing unit to 

20 an edge enhancement unit. Figure 14c shows the inverse histogram based 
mapping unit as a pre-processing unit for the edge enhancement unit. Figure 
14d shows the inverse histogram-based mapping unit in parallel to the edge 
enhancement unit. Figure 14e shows the two inverse histogram-based 
mapping units, one in the original image branch used as a pre-processing unit 

25 for a low pass filter, and the other one in a parallel branch as a pre-processing 
unit for the edge enhancement unit. Other inverse histogram-based mapping 
arrangements are possible in which the method comprises applying the 
image's statistical data dependent pixel mapping function to the image pixels 
in such a way that image contrast is reduced in signal ranges containing a 

30 large number of pixels and increased in other signal ranges. 
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Accordingly reference should be made to the appended claims and other 
general statements herein rather than to the foregoing specific description as 
indicating the scope of invention. 

Furthermore, each feature disclosed in this specification (which term includes 
the claims) and/or shown in the drawings may be incorporated in the Invention 
independently of other disclosed and/or illustrated features. In this regard, the 
invention includes any novel features or combination of features disclosed 
herein either explicitly or any generalisation thereof irrespective of whether or 
not it relates to the claimed invention or mitigates any or all of the problems 
addressed. 

The appended abstract as filed herewith is included In the specification by 
reference. 
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Claims 

1 . A method for image enhancement of a digital image comprising a set of 
image pixels, the method comprising applying an inverse histogram-based 

5 mapping function to the image pixels. 

2. A method according to claim 1, wherein formation of the inverse 
histogram mapping comprises 

constructing a histogram of said image pixels, 
10 developing an inverse histogram from said histogram, 

forming a cumulative inverse histogram from said inverse histogram, 
deriving an inverse mapping function of said inverse histogram 
mapping pixel values based on said inverse mapping function. 

15 3. A method according to claim 1, wherein formation of the inverse 

histogram mapping comprises 

constructing a histogram of image pixel values, 

forming a cumulative histogram from said histogram, 

deriving a mapping function from the cumulative histogram, 
20 forming an inverse mapping function from said derived mapping function, and 

mapping pixel values using the inverse histogram-based pixel mapping 

function. 

4. A method according to any preceding claim, wherein the. 'inverse 
25 histogram based mapping function is applied to the image pixels prior to or 

after applying an edge enhancement unit. 

5. A method according to any of claims 1 to 4, wherein the inverse 
histogram based pixel mapping function is applied in an original signal branch 

30 and an edge enhancement unit is applied in a branch parallel to the original 
signal branch. 
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6. A method according to any of claims 1 to 5, wherein the inverse 
histogram based pixel mapping function is applied in an original signal branch 
and in a branch parallel to the original signal branch having an edge 

5 enhancement unit. 

7. Image processing means for enhancing a digital image comprising a 
set of image pixels, the processing means comprising means for applying an 
inverse histogram mapping function to the image pixels. 

10 

8. A portable radio communication device including image processing 
means for enhancing a digital image comprising a set of image pixels, the 
processing means comprising means for applying an inverse histogram 
mapping function to the image pixels. 

15 

9. A computer program product having a computer usable medium having 
computer readable program code embodied therein for image enhancement 
of a digital image comprising a set of image pixels, the code comprising 
applying an inverse histogram-based mapping function to the image pixels. 

20 

10 A method for image enhancement of a digital video image comprising a 
set of image pixels, the method comprising applying, in a loop of an edge 
enhancement unit that is parallel to an original image branch, a histogram- 
based pixel mapping function to said set of image pixels prior to applying a 
25 highpass filter in said loop. 



Abstract 



This invention provides a method and associated device an inverse histogram 
based pixel mapping step is combined with an edge enhancement step such 
as unsharp masking. In such an arrangement the inverse histogram based 
pixel mapping step improves the performance of the unsharp masking step, 
serving to minimise the enhancement of noise components while desired 
signal components are sharpened. 
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